I. Motion Parallax
Motion parallax stereoscopic display systems are currently enabled in products such as the Learndo 3D terminal, the zSpace terminal, and EON Reality's Ibench and Imobile devices. Motion parallax allows a virtual object to appear stationary in real space when an observer changes perspective. Thus, even though an observer's head is moving, a virtual object appears in a fixed space and the observer can “look around” surfaces of the virtual object. Motion parallax enhances the natural user interface experience afforded by the mixing virtual and real objects in user interface and other applications. FIG. 1 shows an example of a prior art motion parallax stereoscopic display system. An observer in a first position 100A is observing a stereoscopic display 102 projecting a stereoscopic image of a first object image 104 located front and center relative to the display 102 and a second object image 106 located behind and on the left side relative to the display 102. A tracking system 106 has stereo cameras that track the position and motion of the first observer. In order to make object images 104 and 106 appear at their locations, a pair of stereoscopic images 114A and 116A are rendered on the display 102 for the left and right eyes of the observer in a manner known to those familiar with the art. A physical object 120A is operated by the first observer in order to interact with the object images 104 and 106. The position and orientation of the physical object is determined by tracking system 108 and in one example, a stereoscopic extension image 122A of the object is projected by the stereoscopic display. When an intersection of the stereoscopic extension image and the object image 104 or 106 is detected, an appropriate response is generated. One response may be to open an application such as a word processing application. Such a system is described in U.S. Pat. No. 6,243,054 entitled Stereoscopic User Interface Method and Apparatus, issued Jun. 5, 2001 to Michael DeLuca, and is hereby incorporated by reference.
When the first observer moves to a second position 100B, a motion parallax process is used to render the stereoscopic pair of images 114B and 116B to cause the first and second object images 104 and 106 to appear in substantially constant locations relative to the display 102 even though the observer has moved to a different position. Physical object 120B and its stereoscopic extension image 122B are shown to continue to intersect object image 104 because the observer has changed the position and orientation of the real object in response to the position and motion of the observer. Display versions 102A and 102B show a two dimensional representation of the object images 104 and 106 viewed at positions 100A and 100B respectively. Object images 104A and 104B appear larger than object images 106A and 106B because it is closer to the observer. Display 102A shows object image 104A partially obscuring object image 106A because of the observer's position 100A, while display 102B shows object image 104B well to the right of object image 106B because of the observer's position 100B. The system allows for the simultaneous viewing of both virtual object images 104 and 106 and real physical objects, including physical object 120 which is shown as extending beyond the display 102A and 102B as 120A and 120B. Display 102A and 102B also show the stereoscopic extension image 122A and 122B being rendered with motion parallax relative to the physical object.
Motion parallax provides for a natural user interface experience for the first observer. A second observer simply views the images as rendered for the first observer, which results in an experience different from the natural user interface experience of the first observer.
II. Augmented Virtual Reality
Turning to the augmentation of a virtual reality object, head mounted devices are being introduce that allow for augmentation of real objects. For example, U.S. Pat. No. 6,559,813 to DeLuca et al. shows a headset that displays augmented information to an observer also observing a real image. The headset includes a camera system and a projector. The headset camera system monitors the physical objects also viewed by the observer wearing the headset, and the headset projector projects an image that augments the physical objects. In one example, an observer is viewing the Parthenon and a virtual image including additional information regarding the Parthenon is projected by the headset so the observer is able to view both the Parthenon and augmented information regarding the Parthenon. Such devices are currently being introduced including Google's Google Glass, Atheer Labs' headset, and Meta's Space Glasses.
U.S. Pat. No. 6,559,813 also describes an image obstruction generator for blocking portions of a real image. In one example, the obstruction generator may block a bright portion of the real image, which may enhance viewing of other real and/or virtual objects in the field of view of the observer. Products which implement such an obstruction generator covered by the patent are being introduced and include the sunglasses made by Dynamic Eye.
U.S. Pat. No. 6,559,813 also describes a headset that projects a stereoscopic display wherein physical objects, such as a finger or hand of the observer, or a pointer held by the observer may interact with stereoscopic objects projected by the display based upon a determined intersection between the physical object and the virtual object. Headset products which implement intersections between virtual and real objects are being introduced and include the Atheer Labs' headset and Meta's Space Glasses.
Thus, headsets have been provided that produce images that interact with real objects within view of an observer. This is currently being associated with the term “natural user interface”. An observer may be able to view and interact with a stereoscopic image projected by a headset projector as if the projected object image was a real object. However, headsets are incapable of interacting with a virtual object image projected by a stereoscopic display separate from the headset, such a display of a 3D movie, a zSpace terminal, or other stereoscopic projection.
III. Proprioceptive User Interface
Turning now to proprioception, proprioception is a characteristic natural to humans that describes an individual's sense of the relative positions of neighboring parts of their body. For example, one's ability to touch the nose on their face with their eyes closed or one's ability to instinctively operate an automobile without looking at the foot pedals, gear shifter, or steering wheel is attributed to proprioception. Much of proprioception is learned through visual feedback, and once learned, the individual can rely on muscle memory to complete an operation, without requiring visual feedback. For example, after an individual sees the location of automobile foot pedals and sees their feet operating the pedals, the location of the foot pedals is retained in muscle memory and the individual may only need to occasionally look at the pedal locations again to operate the automobile.
When an individual operates a graphical user interface to a computerize system, numerous interface icons may be rendered within the system's display area to allow a user to select which functionality of the computerized system is desired. A typical user interface home screen may have dozens or more interface icons for accessing a network, the Internet, often used applications, hard drive directories, file deletion functionality and other often used files. These icons have the benefit of allowing for quick access to processes associated with icons, however they have a detriment in that they obscure the display area for other relevant information. For example, a computer automated design (CAD) application may be optimal if it occupies the entire area of a display, but if the user also needs occasional Internet access, either the display area of the CAD application while using the application is reduced to allow for viewing of the Internet icon, thereby diminishing the CAD application experience, or the user wanting to access the Internet while using the CAD application must be reduced or the display area of the CAD application window to reveal the Internet access icon, and then select the icon, thereby taking extra process steps and operating time, ultimately diminishing the user interface experience.
A user of such a computer system moves frequently during its operation, and the location of the icons relative to the operator's body varies widely, as such applications are not able to use proprioception to resolve the above stated problems. Furthermore, stereoscopic user interfaces such as 3D terminals made Learndo 3D, zSpace, and EON Reality and the 3D terminal described in U.S. Pat. No. 6,559,813, describe detecting an intersection between a virtual object and a real object as perceived by an observer. Additionally, headsets from Atheer Labs and Meta and the 3D headset described in U.S. Pat. No. 6,559,813, also describe detecting an intersection between a virtual object and a real object in a space relative to an observer. Furthermore, portable display devices such as cellphones, tablets, eBooks, and video games are able to move through a space relative to an observer while rendering augmented images relative to the space. Such devices ignore the natural human characteristic of proprioception in their user interfaces.